Add detector-to-decision layer for region-based VLM inference - #37

Summary

This PR introduces a new laya.regions module that bridges object detectors and the VLM decision model, enabling two key workflows: selecting one region from a detector's output, and mapping decisions across multiple regions via cropping.

Key Changes

• New laya/regions.py module (511 lines):

  • Region dataclass representing detections with pixel boxes, optional labels, scores, and masks
  • Detector adapters (as_regions, from_ultralytics, from_supervision, from_sam, from_omniparser, from_hf_pipeline) supporting YOLO, RT-DETR, RF-DETR, SAM, OmniParser, and transformers pipelines
  • select() function for one-choice questions over regions with Set-of-Mark visualization and knockout rounds for large region counts
  • map_regions() function for applying questions to each region's crop
  • stream() function for processing frame sequences with background detection
  • RegionPipeline class wrapping the above for convenient chaining
  • draw_marks() for numbered region visualization

• Comprehensive test suite (tests/test_regions.py, 220 lines):

  • Tests for all detector adapters
  • Tests for region cropping, selection, mapping, and streaming
  • End-to-end test with SmolVLM-256M checkpoint

• Benchmark probes for measuring effectiveness:

  • benchmarks/marks_probe.py: Synthetic Set-of-Mark readability test (digit and colored box recognition)
  • benchmarks/crop_probe.py: COCO ground-truth crop verification and 8-way classification
  • benchmarks/zoom_probe.py: Oracle zoom ceiling on V*Bench small objects
  • benchmarks/zoom_search_probe.py: Glance-then-zoom search strategies

• Documentation (site-docs/how-to/detector-regions.md):

  • Usage guide for plugging in detectors
  • Examples of select() and map_regions() workflows
  • Explanation of Set-of-Mark visualization and knockout rounds

• Module exports: Added Region and RegionPipeline to laya/__init__.py

Notable Implementation Details

• Duck-typed detector support: No hard dependencies on detector libraries; adapters work via attribute inspection
• Knockout algorithm: Handles >10 regions by grouping into rounds, with winners advancing until one remains
• Crop padding: Configurable padding and minimum size for region crops to ensure readable input
• Format versioning: REGIONS_FORMAT_VERSION tracks changes to mark drawing and option formatting
• Streaming with background detection: stream() runs detection on a background thread while the decision model processes the current frame, with optional frame dropping for real-time constraints
• Calibration caveat: Documentation notes that no released checkpoint was trained on marked regions or detector crops, so accuracy must be measured per task

https://claude.ai/code/session_01XCUwTdDJhkMRhS21sZ8x8H